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EXECUTIVE  SUMMARY 


The  goals  of  the  Susceptibility  Model  Assessment  and  Range  Test  (SMART)  Project  are 
to  develop  an  efficient  process  for  the  verification  and  validation  (V&V)  and 
configuration  management  (C/M)  of  aircraft  susceptibility  models  and  to  facilitate  their 
quick  and  cost  effective  accreditation  by  applying  this  new  approach  to  five  models 
frequently  used  in  survivability  analyses  supporting  acquisition  decisions.  The  credibility 
assessment  process  thus  defined  generates  various  reports  that  provide  a  model  user 
with  enough  information  on  existing  verification,  validation  (V&V),  and  configuration 
management  (C/M)  data  to  permit  accreditation  with  a  minimum  of  additional  effort.  To 
make  these  reports  most  beneficial  to  accreditation  proponents,  the  SMART  Project 
undertook  a  study  of  accreditation  requirements. 

The  study  involved  a  review  of  existing  instructions  and  directives  that  specify 
accreditation  requirements,  as  well  as  interviews  with  personnel  engaged  in  all  aspects 
of  model  development  and  use,  from  policy  makers  to  model  users.  Analysis  of  this 
information  focused  on  two  principal  aspects  of  typical  accreditation  decisions: 
accreditation  processes  and  information  requirements.  The  analysis  of  accreditation 
processes  is  presented  in  a  companion  report.  This  report  presents  findings  on  the 
information  required  to  accredit  models. 

The  factors  most  frequently  mentioned  as  being  important  for  determining  model 
suitability  for  a  particular  application  were: 

•  Clear  acceptance  criteria; 

•  Comparison  of  model  assumptions  and  limitations  with  the  characteristics  and 
boundaries  of  the  specific  application; 

•  Model  usage  history  and  past  V&V  results; 

•  Results  of  V&V  performed  specifically  for  the  intended  application; 

•  Effective  configuration  management. 

Of  these  factors,  the  need  for  clear  acceptance  criteria  on  which  to  base  a  decision  was 
judged  to  be  most  critical.  These  criteria,  in  turn,  were  said  to  be  critically  dependent  on 
developing  appropriate  measures  of  effectiveness  for  the  particular  application.  Even 
with  an  accredited  model,  it  was  strongly  felt  that  a  study  could  produce  erroneous 
results  if  inappropriate  measures  of  effectiveness  were  used,  even  with  ostensibly 
acceptable  models. 

The  study  findings  led  to  recommended  changes  to  SMART  documentation  products.  A 
detailed  listing  of  information  requirements  supporting  accreditation  decisions  derived 
from  the  documents  and  interviews  was  compared  to  the  current  contents  of  the 
SMART  products.  Items  not  currently  included  in  the  SMART  products  were  noted,  and 
recommendations  for  specific  additions  were  developed  and  are  presented  herein. 

A  principal  finding  of  this  study  is  that  all  the  information  produced  by  the  SMART 
process  is  important  to  most  accreditation  decisions.  Analysis  also  shows  that  the 
current  SMART  process  produces  all  the  most  frequently  cited  information  and,  if  the 
recommended  additions  are  incorporated,  will  contain  over  70%  of  all  the  information 
that  was  mentioned  in  this  study.  Those  information  requirements  not  included  were 


only  mentioned  once  or  twice,  or  are  redundant  with  other  V&V  information  being 
produced  by  the  SMART  process. 

An  analysis  of  the  information  collected  in  this  study  reinforces  the  need  for  a  standing 
library  of  up-to-date  V&V  information  for  frequently  used  models  and  simulations  (M/S). 
With  a  library  of  such  information  each  model  user  need  perform  only  those  V&V  tasks 
tailored  to  changed  portions  of  the  model(s)  or  applicable  to  unique  scenarios.  If  this 
additional  information  is  then  fed  back  into  the  V&V  library,  each  subsequent  user  has  a 
greater  body  of  information  on  which  to  base  accreditation.  The  products  being 
produced  by  the  SMART  Project  form  a  ibrary  of  standard  data  that  serves  to  improve 
the  quality  of  the  V&V  information  available  and  increases  the  efficiency  of  the 
accreditation  process. 

Given  the  current  DoD  and  service  emphasis  on  the  use  of  accredited  M/S  in  system 
acquisition,  considerable  interest  has  been  exhibited  in  various  VV&A  approaches  and 
the  costs  of  each.  To  address  these  concerns,  the  SMART  process  has  been  divided 
into  increments  that  can  be  performed  sequentially.  Each  incremental  task  yields  a 
product  that  adds  to  the  overall  library  of  V&V  information  on  a  model.  Use  of  the 
SMART  products  in  accrediting  ESAMS,  ALARM  or  RADGUNS  for  a  specific  application 
will  save  the  typical  user  a  significant  amount  of  time  and  money  that  would  normally  be 
spent  in  collecting  V&V  information  for  each  accreditation  decision.  Adoption  of  the 
SMART  process  as  the  V&V  methodology  by  all  users  of  a  model  will  facilitate  the  task 
of  keeping  the  library  of  V&V  data  current  as  the  model  undergoes  revision.  Expansion 
of  the  SMART  domain  to  other  M/S,  and  adoption  of  the  incremental  approach  for 
applying  the  process  will  benefit  a  correspondingly  wider  community  of  users.  Use  of 
the  SMART  process  in  producing  application-specific  V&V  results  in  support  of  future 
accreditation  decisions  will  also  minimize  the  amount  of  variability  that  exists  in  the 
quality  and  scope  of  V&V  that  is  done. 
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1.  BACKGROUND  AND  PURPOSE 


The  goal  of  the  Susceptibility  Model  Assessment  and  Range  Test  (SMART)  Project  is  to 
develop,  document,  establish,  and  transition  a  process  for  assessing  the  credibility  of 
models  and/or  simulations  (M/S)  used  in  the  aircraft  survivability  discipline.  The 
impetus  for  this  goal  is  to  support  acquisition  decisions  (e.g.,  Cost  and  Operational 
Effectiveness  Analyses  [COEAs]  and  Defense  Acquisition  Board  [DAB]  reviews);  test 
planning  and  analysis  during  the  development  test  (DT)  and  operational  test  (OT) 
phases  of  a  program;  mission  planning;  and  many  other  applications  requiring  analysis 
of  the  combat  effectiveness  of  military  aircraft.  The  M/S  credibility  assessment  process 
involves  elements  of  verification,  validation  (V&V)  and  configuration  management  (C/M). 
Implementation  of  this  process  yields  the  core  information  required  for  accreditation  of 
survivability  M/S  used  in  analyses.  SMART  is  producing  baseline  V&V  and  C/M  infor¬ 
mation  on  a  range  of  aircraft  survivability  M/S,  and  building  an  accreditation  support 
data  base  that  allows  M/S  users  to  learn  from  and  build  upon  the  work  of  previous  M/S 
users.  In  this  way,  M/S  accreditation  requirements  in  support  of  acquisition  are  more 
easily  identified,  and  accreditation  is  more  easily  (and  cheaply)  performed. 

To  maximize  the  utility  of  SMART  V&V  and  C/M  documentation  to  the  accreditation  pro¬ 
cess,  it  became  necessary  to  understand  what  information  is  typically  required  to  sup¬ 
port  accreditation.  Since  a  single  source  of  accreditation  information  requirements  did 
not  exist,  SMART  commissioned  a  study  to  identify  the  requirements  of  the  various  ac¬ 
tivities  that  use  and  accredit  M/S  across  the  services.  It  was  intended  that  comparison 
between  these  requirements  and  the  current  SMART  products  would  provide  useful  in¬ 
sight  into  what  information  should  be  added  to  make  SMART  products  most  useful  to 
those  involved  in  actual  accreditation  decisions. 

To  establish  a  common  understanding  of  the  purpose  and  requirements  of  this  study,  it 
is  necessary  to  understand  the  meaning  of  the  terms  verification,  validation,  configura¬ 
tion  management,  and  accreditation.  Although  the  definition  of  these  terms  varies  from 
user  to  user,  the  community  has  generally  adopted  the  definitions  formulated  by  the 
Military  Operations  Research  Society  (MORS): 

Verification  is  the  process  of  determining  the  degree  to  which  an  M/S  accurately 
represents  the  developer's  conceptual  description  and  specifications.  Verification 
entails  a  logical  evaluation  (evaluating  inputs,  outputs  and  code  to  assess 
whether  the  algorithms,  equations,  and  results  are  logical)  and  a  code  verification 
(assessing  the  actual  computer  code  using  a  "reverse  engineering"  approach  to 
determine  whether  it  accurately  reflects  the  developer’s  specifications  for  algo¬ 
rithms,  equations,  and  operational  capability). 

Validation  is  the  process  of  determining  the  degree  to  which  an  M/S  is  an  accu¬ 
rate  representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of 
the  M/S.  It  is  not  an  absolute  statement  of  M/S  fidelity,  but  an  ongoing  process  of 
establishing  and  increasing  the  accuracy  and  confidence  levels  of  the  M/S  for 
particular  applications.  This  process  requires  expert  assessment  of  the  informa¬ 
tion  used  in  basic  M/S  relationships,  sensitivity  analysis  to  determine  the  parame¬ 
ters  and  algorithms  of  key  importance  to  M/S  results,  and  comparison  of  M/S 


1 


predictions  with  real  world  test  data.  In  many  cases,  the  term  “validation”  also 
applies  to  the  input  data  that  is  used  in  M/S  runs. 

Configuration  Management  is  the  process  of  applying  technical  and  administra¬ 
tive  oversight  to  identify,  document,  and  control  the  functional  requirements  and 
capabilities  of  M/S,  control  changes  to  their  capabilities,  and  document,  report 
and  distribute  these  changes  to  users  in  a  timely  fashion. 

Accreditation  is  the  official  determination  that  a  particular  M/S  is  acceptable  for  a 
particular  application.  It  includes  the  judgment  that  the  expected  accuracy  and 
confidence  limits  of  the  M/S  are  adequate  for  the  intended  purpose.  It  also  as¬ 
sumes,  implicitly  or  otherwise,  that  some  form  of  verification,  validation,  and  con¬ 
figuration  management  (as  described  above)  has  occurred,  and  that  it  is  suffi¬ 
cient  for  the  purpose  at  hand. 

Other  terminology  used  in  this  report  reflects  the  terminology  used  by  various  sources 
from  which  this  information  was  drawn.  Different  sources  used  the  terms  model,  model 
and  simulation  (M&S),  and  model  and/or  simulation  (M/S)  to  mean  the  same  thing. 
Since  some  sources  are  quoted  in  this  report,  these  terms  are  used  interchangeably 
depending  on  the  source  under  discussion. 

2.  STUDY  APPROACH 

Information  for  this  study  was  collected  through  personal  interviews  and  document  re¬ 
views.  Key  individuals  in  the  M/S  community  were  contacted  to  determine  the  key  DOD 
and  service  organizations  and  activities  that  use  survivability  M/S  and  perform  verifica¬ 
tion,  validation  and  accreditation  (VV&A)  in  conjunction  with  their  use.  A  list  of  these  or¬ 
ganizations  was  developed  and  points  of  contact  at  each  organization  were  identified 
(see  Appendix  A).  An  interview  guide  was  prepared  to  ensure  that  all  critical  questions 
and  issues  were  addressed.  The  interview  guide  is  contained  in  Appendix  B. 

These  various  organizations  on  the  list  were  visited  to  interview  those  persons  who 
perform  V&V,  develop  VV&A  policies,  or  approve  and  accredit  M/S  as  part  of  their  or¬ 
ganizational  function.  Besides  questioning  the  interviewee  using  the  prepared  guide, 
any  written  material  that  addressed  VV&A  guidelines,  and  established  policy,  or  that 
documented  actual  VV&A  efforts,  were  obtained.  Information  on  other  suggested  points 
of  contact  was  also  requested,  thereby  broadening  the  interview  base  as  wide  as  pos¬ 
sible.  Summary  notes  documenting  the  substance  of  each  interview  were  prepared  and 
provided  to  each  interviewee  for  review  and  comment.  This  practice  ensured  that  inter¬ 
view  statements  were  clearly  understood  and  that  no  bias  was  introduced  into  the  inter¬ 
view  summaries.  Summaries  of  the  interviews  are  provided  as  Appendix  C.  They  are 
organized  in  the  order  shown  on  the  interview  list. 

These  interview  summaries,  along  with  the  documentation  collected,  were  reviewed  to 
identify  the  types  of  information  that  were  used  to  support  selection  of  particular  model 
and  justify  an  accreditation  decision  for  a  study.  A  listing  of  information  critical  to  ac¬ 
creditation  decisions  was  prepared.  Due  to  the  differences  in  terminology  used  by  dif¬ 
ferent  individuals  and  services,  common  terms  were  developed  to  represent  similar  in¬ 
formation  elements.  A  matrix  was  then  prepared  showing  which  elements  from  the  list 
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had  been  identified  as  important  to  each  accrediting  agency  or  activity  and  how  fre¬ 
quently  they  were  identified  as  being  important.  The  most  frequently  cited  items  in  that 
matrix  were  then  compared  to  the  information  elements  of  the  various  reports  that  are 
produced  from  the  SMART  V&V  and  C/M  processes.  Any  disparities  were  noted.  A  list 
of  important  accreditation  elements  that  were  lacking  in  the  SMART  products  was  iden¬ 
tified  for  recommended  incorporation  into  those  products. 

In  compiling  information  requirements  that  support  accreditation,  a  wide  disparity  was 
noted  in  the  scope  and  depth  of  these  requirements.  This  disparity  paralleled  (and  was 
apparently  related  to)  the  wide  variety  of  opinions  and  ideas  regarding  proper  processes 
and  procedures  used  to  accredit  a  model.  Some  examples  of  this  variability  can  be 
seen  in  the  interview  reports  for  the  Navy’s  Operational  Test  and  Evaluation  Force 
(OPTEVFOR)  and  the  Army  Material  Systems  Analysis  Activity  (AMSAA).  OPTEVFOR 
requests  assistance  from  another  Navy  activity,  the  Center  for  Naval  Analysis  (CNA),  to 
conduct  an  accreditation  study  and  to  provide  a  report  that  recommends  whether  or  not 
a  model  should  be  accredited.  There  is  little  guidance  provided  by  OPTEVFOR  on  what 
information  should  be  considered,  or  on  what  criteria  should  be  used  to  make  such  rec¬ 
ommendations.  On  the  other  hand,  AMSAA  has  a  detailed  process  that  entails  a  study 
of  specific  aspects  of  a  model.  Their  study  requirements  are  specified  in  a  briefing 
checklist  that  identifies  specific  V&V  information  requirements,  including:  model  devel¬ 
opment  history;  code  review  results;  documentation  review;  data  certification;  previous 
V&V  results;  specific  validation  results;  configuration  management  provisions;  and  ac¬ 
ceptance  criteria  for  judging  model  acceptability.  This  variability  indicates  that,  in  prac¬ 
tice  at  least,  accreditation  requirements  are  in  the  eye  of  the  beholder. 

Although  the  initial  objective  of  this  study  was  to  determine  the  information  most  fre¬ 
quently  used  to  accredit  models,  the  focus  of  the  study  was  expanded  when  it  became 
apparent  that  the  emerging  accreditation  policies  differed  markedly  from  the  current 
practices.  These  differences  had  to  be  understood  to  determine  if  the  current  informa¬ 
tion  requirements  would  be  likely  to  change  as  new  policies  are  implemented. 
Therefore,  a  more  detailed  analysis  of  the  various  accreditation  processes  was  under¬ 
taken.  The  discussion  of  the  different  accreditation  practices  and  policies  establishes 
the  framework  for  analyzing  the  current  information  requirements  and  determining  which 
elements  are  most  important.  The  results  of  this  policy  and  practice  analysis  are  pre¬ 
sented  in  a  companion  report  entitled  “A  Comparative  Analysis  of  Tri-Service 
Accreditation  Policies  and  Practices,”  Volume  I  of  Report  #  JTCG/AS-93-SM-20. 

3.  INTERVIEW  FINDINGS 

The  most  significant  points  addressed  by  persons  from  several  different  organizations 
concerned:  1)  resource  constraints;  2)  lack  of  common  V&V  techniques  and  ap¬ 
proaches;  and  3)  problems  associated  with  selecting  a  model  for  a  study.  The  major 
points  gleaned  from  these  summaries  are  highlighted  in  the  following  paragraphs. 
Detailed  interview  summaries  are  presented  in  Appendix  C. 

3.1  Resource  Constraints 


Several  interviews,  most  notably  those  at  AMSAA  and  CNA,  mentioned  VV&A  cost  as 
an  important  consideration.  The  value  added  by  collecting  extensive  information  to 
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support  accreditation  must  be  weighed  against  the  cost  and  personnel  required  and  the 
importance  of  the  decision.  Concerns  were  also  expressed  about  the  cost  of  formal 
configuration  management  procedures,  the  cost  of  applying  the  SMART  V&V  process  to 
other  models,  and  the  need  to  collect  data  on  actual  costs  of  performing  V&V.  (SMART 
is  addressing  this  in  its  FY94  tasking.) 

Other  interviewees  pointed  out  that  there  is  seldom  enough  time  to  carry  out  a  formal 
model  accreditation  for  a  particular  application  since  the  study  results  are  often  required 
in  the  matter  of  a  few  weeks.  The  organizations  faced  with  this  type  of  problem  are  the 
Army  Deputy  Chief  of  Staff  for  Operations  (DCSOPS),  the  Naval  Air  Systems 
Command,  Warfare  Analysis  Division  (NAVAIR-526),  OPTEVFOR,  and  the  Air  Force 
Aeronautical  Systems  Command  (ASC).  Occasionally,  the  lack  of  time  forces  analysts 
to  use  models  that  are  not  really  suited  to  the  study  but  are  either  the  only  ones  avail¬ 
able  or  the  only  ones  with  which  the  analyst  is  familiar. 

These  concerns  about  the  cost  and  time  required  to  collect  extensive  information  about 
a  model  stem  from  the  lack  of  common  information  requirements  to  support  accredita¬ 
tion.  If  there  were  common  information  requirements,  a  library  of  up-to-date  V&V  infor¬ 
mation  could  be  established  for  the  most  frequently  used  models.  Such  a  library  should 
contain  a  standard  set  of  information  elements  that  are  most  frequently  used  by  a 
majority  of  model  users.  Through  this  analysis  of  accreditation  information  require¬ 
ments,  the  SMART  project  intends  to  define  a  common  set  of  information  requirements 
and  to  establish  a  library  of  V&V  data  that  fulfills  most  of  these  requirements. 

3.2  V&V  Techniques 

Several  V&V  techniques  were  mentioned  as  being  valuable  in  supporting  accreditation. 
Although  almost  all  interviewees  recognized  the  desirability  of  performing  extensive 
verification  code  checks  and  in  depth  comparisons  between  model  results  and  real 
world  data,  funding  and  time  constraints  usually  preclude  such  extensive  V&V.  Instead, 
many  model  users  turn  to  other  less  costly  methods  that  are  also  less  beneficial.  Such 
methods  typically  include  reviews  of  past  model  usage  and  VV&A  results,  face 
validation,  and  some  comparisons  between  models  (commonly  referred  to  as 
“benchmarking.”).  Among  those  who  use  these  methods  are  the  Army  Operational  Test 
and  Evaluation  Command  (OPTEC),  the  Army  Aviation  and  Troop  Command  (ATCOM), 
AMSAA,  the  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC),  ASC  and 
NAVAIR-526.  A  common  view  is  that  past  usage,  coupled  either  with  reviews  by 
subject  matter  experts  (SMEs)  or  comparisons  between  models,  are  sufficient  to  justify 
model  selection  and  use.  Detailed  comparisons  between  the  model  results  and  test 
data  are  made  by  organizations  engaged  in  test  programs  such  as  AFOTEC  and 
OPTEC.  These  comparisons  are  made  in  parallel  with  the  analysis  being  performed 
using  the  model.  Several  interviewees  specifically  mentioned  the  value  of  understand¬ 
ing  the  assumptions  and  limitations  and  performing  a  sensitivity  analysis. 

AMSAA  and  OPTEC  have  employed  automated  (Computer  Aided  Software 
Engineering,  or  CASE)  tools  with  some  success.  However,  the  analyst  needs  a  good 
understanding  of  the  tools  available  and  how  they  are  best  applied.  According  to  CNA, 
the  best  use  of  these  tools  is  to  document  the  software;  current  model  users  are  gen¬ 
erally  hindered  by  poor  documentation. 
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A  prime  factor  mentioned  by  a  number  of  persons  is  the  importance  of  well-qualified 
analysts.  CNA,  Air  Force  Studies  and  Analysis  Agency  (AFSAA),  ATCOM,  and  some 
divisions  in  ASC  rely  on  good  analysts  to  understand  the  models  they  use  and  ensure 
that  they  will  produce  reliable  results.  They  feel  that  an  analyst  who  uses  a  few  models 
on  a  regular  basis  knows  model  limitations  well  and  is  best  suited  to  determine  if  the 
model  is  applicable  to  a  particular  study. 

One  other  concern  related  to  V&V  techniques  is  data  validation.  Many  organizations 
addressed  this  issue  in  the  documentation  provided  as  part  of  the  interview.  Both 
OPTEC  and  ASC  personnel  stressed  that  study  validity  was  directly  related  to  using 
valid  data  from  an  approved  source  for  a  study. 

The  variety  of  V&V  techniques  as  well  as  the  other  factors  mentioned  introduce  a  po¬ 
tentially  great  amount  of  variability  into  the  depth  and  breadth  of  the  work  done  to  sup¬ 
port  an  accreditation  decision.  This  variability  can  be  mitigated  through  the  develop¬ 
ment  and  use  of  standard  sets  of  accreditation  information  requirements,  as  well  as 
standard  processes  to  generate  supplemental  information.  Agreement  on  a  common 
set  of  information  elements,  as  defined,  for  example,  by  this  study,  will  reduce  some  of 
the  variability  in  accreditation  justification.  Use  of  the  SMART  products  as  a  V&V  base¬ 
line  coupled  with  adoption  of  the  SMART  process  to  generate  supplemental  V&V  infor¬ 
mation  will  further  reduce  this  variability. 

3.3  Model  Selection  Concerns 


One  point  brought  out  by  both  the  OPTEC  and  the  Air  Warfare  Center  (AWC)  intervie¬ 
wees  seems  obvious  but  is  so  important  that  it  bears  repeating.  Using  an  accredited 
model  is  not  the  only  key  to  obtaining  valid  study  results.  The  correct  measures  of  per¬ 
formance  (MOPs)  or  measures  of  effectiveness  (MOEs)  for  the  study  must  be  chosen 
before  the  study  begins.  The  use  of  inappropriate  MOPs  or  MOEs  can  lead  to  erro¬ 
neous  study  results,  even  if  the  model  used  to  support  the  study  is  an  accurate  repre¬ 
sentation  of  “reality.”  Appropriate  MOEs  or  MOPs  should  be  derived  from  the  study  ob¬ 
jectives  early  on.  This  point  cannot  be  overemphasized  in  light  of  the  relationship  be¬ 
tween  study  MOEs  and  model  acceptance  criteria,  a  point  that  will  be  amplified  in  later 
paragraphs.  OPTEC  has  a  guide  for  choosing  their  M/S  acceptance  criteria,  and 
AMSAA  has  included  M/S  acceptance  criteria  as  a  major  item  in  their  accreditation  re¬ 
views. 

The  stated  need  to  identify  appropriate  MOEs  or  MOPs  and  from  them  to  define  proper 
acceptance  criteria  for  a  model  points  to  the  benefits  of  having  a  library  of  V&V  informa¬ 
tion  that  can  be  compared  to  these  acceptance  criteria  during  the  model  selection 
process.  This  concept  of  acceptance  criteria  and  a  library  of  V&V  information  requires 
that  validation  results  should  be  stated  in  terms  of  model  limitations  or  objective  state¬ 
ments  regarding  correlation  fidelity.  Only  with  objective  statements  of  model  fidelity  can 
a  user  make  an  independent  comparison  between  the  model  capabilities  and  the  accep¬ 
tance  criteria  for  a  particular  application.  A  methodology  for  developing  appropriate 
MOEs  and  MOPs  and  defining  acceptance  criteria  is  discussed  further  in  the  accompa¬ 
nying  volume  to  this  report  entitled  “A  Comparative  Analysis  of  Tri-Service  Accreditation 
Policies  and  Practices,”  Volume  I  of  Report  #  JTCG/AS-93-SM-20. 
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Another  issue  mentioned  by  a  large  number  of  interviewees  was  configuration  man¬ 
agement.  Knowledge  of  all  changes  to  a  model  that  make  it  different  from  the  one  de¬ 
scribed  in  the  model’s  baseline  documentation  was  deemed  essential  to  determining  its 
suitability  for  a  particular  application.  Many  also  expressed  concerns  over  the  time  it 
takes  to  get  proposed  and  required  changes  incorporated,  tested,  and  distributed  to  the 
users. 

The  expressed  concerns  about  configuration  management  point  out  the  importance  of 
being  able  to  relate  V&V  information  to  a  particular  version  of  a  model.  This  concern  is 
the  basis  for  the  SMART  project  efforts  to  develop  common  configuration  management 
requirements  and  guidelines  that  are  linked  to  the  V&V  process,  so  that  the  user  has  a 
means  of  relating  the  V&V  results  to  all  applicable  model  versions. 

4.  INFORMATION  REQUIREMENTS  ANALYSIS 

The  preceding  section  highlighted  the  important  elements  extracted  from  the  interview 
summaries.  To  ensure  that  the  SMART  products  contain  most  or  all  the  information 
needed  by  study  analysts  gathering  accreditation  support  information,  a  listing  of  all 
data  items  mentioned  by  any  of  the  interviewees  was  developed.  A  matrix  showing 
these  information  elements  is  shown  in  Table  1.  Where  different  organizations  cited  an 
information  element  that  appeared  to  be  similar  to  one  cited  by  another  organization 
albeit  using  different  terminology,  a  common  term  was  defined  to  cover  both  information 
elements.  Furthermore,  in  identifying  information  elements,  each  activity  or 
organization  cited  the  various  elements  in  different  ways.  Some  of  the  elements  were 
cited  as  requirements;  others  were  listed  as  examples  of  what  might  be  appropriate;  still 
others  were  cited  as  actually  used  in  a  recent  accreditation.  A  key  to  the  matrix  entries 
is  given  in  Table  2.  Definitions  of  the  terms  used  to  describe  the  information  elements 
in  the  matrix  are  contained  in  Appendix  D.  An  understanding  of  these  terms  is  essential 
to  the  following  discussion. 

The  organization  of  the  information  elements  in  the  matrix  is  somewhat  subjective. 
Although  verification  and  validation  are  done  in  support  of  accreditation,  most  of  the  in¬ 
terviewees  and  documents  identified  these  functions  as  separate  activities.  Verification 
and  validation  are  generally  the  responsibility  of  analysts,  whereas  accreditation  is  per¬ 
formed  by  an  executive  agent.  Because  these  functions  are  generally  treated  sepa¬ 
rately  by  M/S  users,  they  are  listed  and  discussed  separately  in  this  report.  The  infor¬ 
mation  requirements  for  accreditation  are  divided  into  three  categories:  those  that  sup¬ 
port  decisions  for  a  "class  of  applications";  those  that  support  decisions  for  a  specific 
application;  and  those  that  are  needed  for  both  types  of  accreditation. 
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TABLE  1  ACCREDrTATION  REQUIREMENTS  MATRIX  (SHEET  2) 
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TABLE  2  ACCREDITATION  REQUIREMENTS  MATRIX  DATA  ENTRY  KEY 


CODE 

DEFINITION 

A 

Addressed  -  Information  on  the  indicated  item  is  requested 

E 

Example  -  Although  not  specifically  cited  as  an  accreditation  item, 
the  indicated  item  is  cited  as  an  example  of  what  is  expected 

1 

Implied  -  Although  not  specifically  cited  as  an  accreditation  item,  the 
indicated  item  is  necessary  to  meet  some  other  specified  require¬ 
ment. 

P 

Performed  -  Item  was  included  in  an  actual  VV&A  report  for  a  spe¬ 
cific  model  or  simulation. 

R 

Required  -  The  indicated  item  is  required  for  accreditation. 

4.1  Information  Elements  Supporting  Verification 

Since  "verification"  is  the  process  of  determining  that  the  model  accurately  represents 
the  developer’s  intentions,  and  that  this  representation  meets  identified  requirements, 
the  critical  factors  are  understanding  the  requirements  and  knowing  how  the  developer 
intended  to  meet  (or  actually  met)  them.  Any  requirements  documents  or  conceptual 
specifications,  as  well  as  the  various  manuals  describing  model  operation  and  use,  are 
the  best  sources  of  this  information.  The  availability  and  adequacy  of  these  materials  is 
one  of  the  factors  deemed  most  important  to  an  accreditation  decision,  because  they 
form  the  basis  for  code  checks  and  because  they  facilitate  ready  identification  of  M/S 
limitations  and  constraints. 

The  two  principal  aspects  of  verification  identified  by  the  study  participants  were  logical 
verification  and  code  verification.  Logical  verification  consists  of  those  reviews  meant  to 
ensure  the  accuracy  and  consistency  of  a  model’s  assumptions,  equations,  and  algo¬ 
rithms.  A  thorough  review  of  the  assumptions  and  limitations  was  considered  essential 
to  understanding  a  model.  Users  wanted  to  know  both  explicit  and  implicit  assumptions 
and  limitations  of  the  model  in  order  to  understand  how  best  to  use  it.  These  assump¬ 
tions  and  limitations,  along  with  the  various  model  descriptions,  provide  an  insight  into 
the  developer's  intended  design,  form  the  basis  for  any  verification  effort,  and  are  critical 
to  determining  that  a  given  model  is  suitable  for  use  in  a  particular  application. 

The  other  commonly  cited  aspect  of  logical  verification  was  a  review  of  the  basic  algo¬ 
rithms  to  ensure  that  they  adequately  portray  reality  and  are  applied  in  the  appropriate 
regimes.  The  parametric  boundaries  beyond  which  the  mathematical  approximations 
no  longer  reflect  real  world  phenomena  can  be  deduced  from  a  “walk-through”  of  the  al¬ 
gorithms  and  equations.  For  example,  the  algorithms  used  to  represent  normal  aircraft 
cruise  performance  should  be  checked  to  ensure  that  they  are  not  employed  at  low 
speed  and  high  angles  of  attack  where  the  linear  assumptions  about  lift  and  drag  are  no 
longer  valid.  These  logical  checks  are  often  performed  by  independent  SMEs  or  inde¬ 
pendent  analysts  and  are  frequently  termed  "peer  reviews"  Such  reviews  are  commonly 
used  as  part  of  verification. 
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Other  reviews  that  are  performed  as  part  of  the  logical  verification  include  documenta¬ 
tion  checks  and  design  logic  checks  by  SMEs.  These  are  done  to  verify  that  the  flow 
diagrams,  model  structure,  and  process  descriptions  are  consistent  and  acceptable 
from  the  perspective  of  system  experts.  Another  review,  included  under  logical 
verification  but  more  closely  related  to  the  input  data  used  in  an  application,  is  a  check 
on  the  consistency  of  data  definitions  between  the  source  data  and  that  specified  for  the 
model  inputs. 

Code  verification  is  a  detailed  and  (often)  laborious  comparison  of  the  code  to  the  model 
software  specifications.  If  the  specifications  do  not  exist  (a  common  circumstance  for 
many  existing  models)  the  verifier  must  develop  them  through  an  analysis  of  the 
existing  manuals  and  through  reverse  engineering  of  the  code,  frequently  in  conjunction 
with  the  original  model  developer.  Such  efforts  are  time  consuming  and  can  be  costly. 
Therefore,  in-depth  code  verification  is  seldom  done  in  support  of  current  accreditation 
decisions. 

One  technique  that  was  cited  as  important  to  verification,  but  also  applies  to  validation, 
was  a  sensitivity  analysis.  It  allows  an  analyst  to  determine  which  input  parameters 
and/or  model  subroutines  have  the  most  significant  effect  on  model  outputs.  It  provides 
a  means  of  determining  that  the  code  is  functioning  properly  for  varying  sets  of  input 
data.  A  sensitivity  analysis  of  model  input  and  output  relationships  can  also  serve  to 
define  hidden  assumptions  and  limitations.  A  sensitivity  analysis  also  provides  the 
basic  information  needed  to  check  that  the  model  is  reacting  to  varying  inputs  in  a 
reasonable  and  mathematically  predictable  manner.  Their  results  also  increase 
confidence  that  the  model  is  sensitive  to  those  parameters  that  tend  to  cause  the 
greatest  perturbations  in  real  world  results.  The  information  resulting  from  the 
sensitivity  analyses  is  also  important  in  reaching  an  accreditation  decision.  The  effect  of 
M/S  outputs  on  the  MOEs  or  MOPs  used  in  the  analysis  can  be  evaluated  in  light  of  the 
sensitivity  analysis  results. 

In  many  cases,  the  results  of  these  sensitivity  analyses  are  used  to  structure  stress 
tests  (i.e.,  test  runs  of  the  model  using  limit  values  for  certain  parameters)  to  study  the 
effects  of  extreme  inputs  on  model  results,  or  to  determine  that  the  model  will  not 
"crash"  under  these  conditions.  Stress  testing  is  time  consuming,  but  is  considered  im¬ 
portant  in  those  studies  where  the  study  boundaries  are  near  or  outside  the  accepted 
model  envelope. 

Because  the  results  of  sensitivity  analyses  give  the  analyst  wide  ranging  insights  into 
the  model  and  its  limitations,  this  technique  is  considered  important  for  generating  ac¬ 
creditation  information.  It  is  an  essential  component  of  in  depth  V&V. 

Finally,  automated  software  tools  (CASE  tools)  were  also  indicated  as  being  useful  to 
the  verification  effort.  Such  tools  can  assist  in  documenting  previously  undocumented 
code,  developing  flow  charts,  and  identifying  portions  of  the  code  most  likely  to  contain 
errors.  Although  not  a  direct  source  of  accreditation  information,  these  tools  are  a  major 
aide  to  performing  in  depth  code  checks  on  M/S. 

4.2  Information  Elements  Supporting  Validation 
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Validation  is  a  comparison  between  test  data  and  model  predictions  when  the  model  is 
run  under  the  same  input  and  environmental  conditions  as  the  test.  In  order  to  validate 
a  model,  a  clear  definition  of  the  real  world  conditions  is  essential.  This  definition  must 
include  criteria  by  which  the  adequacy  of  the  model’s  representation  of  the  real  world 
can  be  judged.  These  criteria  will  typically  include  accuracy  and  sampling  rates 
required  for  statistical  confidence  in  the  comparison  between  test  data  and  M/S 
predictions.  (For  example,  how  closely  must  the  test  data  agree  with  model  predictions 
and  how  frequently  must  data  measurements  be  made  to  generate  confidence  in  this 
result.)  The  definition  of  real  world  conditions  and  application  criteria  are  the  responsi¬ 
bility  of  the  model  user,  and  are  prerequisite  to  evaluating  validation  results. 

Once  the  real  world  application  is  defined  and  understood,  and  the  application  criteria 
have  been  clearly  established,  the  comparisons  between  model  results  and  test  data 
results  are  the  next  most  important  element  of  model  validation.  The  model  user  is  not 
necessarily  forced  into  performing  dedicated  tests  to  make  these  comparisons.  In  fact, 
for  some  classes  of  models  (e.g.,  force  on  force),  adequate  tests  cannot  be  conducted. 
With  a  clear  understanding  of  what  information  is  required  about  the  real  world,  how¬ 
ever,  the  model  validator  can  use  existing  data  from  tests,  operations,  or  even  combat 
as  a  basis  for  comparisons.  The  one  key  factor  in  determining  data  utility  is  adequate 
descriptive  documentation,  which  must  provide  enough  information  to  allow  an 
evaluation  of  whether  the  data  meet  the  defined  real  world  description  criteria. 

Sensitivity  analyses  are  also  important  in  the  validation  effort.  The  model  must  be  com¬ 
pared  to  the  real  world  on  the  edges  of  the  application  envelope.  Sensitivity  analyses 
determine  which  parameters  have  the  greatest  impact,  and  which  are  most  likely  to 
cause  significant  variations  in  the  results  at  the  edge  of  the  envelope. 

Other  validation  techniques  (employed  when  test  data  are  not  available,  when  they  can¬ 
not  be  economically  obtained,  or  when  time  prohibits  an  adequate  data  collection  effort) 
are  "benchmarking"  and  "face  validation”.  Benchmarking  is  the  comparison  between  a 
model’s  output  and  the  outputs  of  other  models  or  simulations,  all  of  which  represent  the 
same  input  and  environmental  conditions.  The  tacit  assumption  is  made  that  two  or 
more  models  will  not  be  equally  “wrong”  in  their  predictions  of  the  outcome  of  a  given 
phenomenon,  and  that  good  correlation  among  and  between  model  results  indicates 
“reasonableness.”  Benchmarking  is  not  included  as  a  technique  in  the  SMART  process. 
However,  the  SMART  accreditation  support  database  (ASD)  will  include  results  of  any 
benchmarking  done  by  other  organizations  in  the  course  of  gathering  their  own 
information  to  support  their  specific  accreditation.  Inputs  regarding  any  VV&A  efforts  by 
all  model  users  will  be  incorporated  in  the  database  and  will  be  available  for  new  model 
users. 

Face  validation  is  the  technique  of  reviewing  the  model,  its  algorithms  and  equations,  by 
a  group  of  subject  matter  experts  to  determine  if  they  appear  logical  and  yield  reason¬ 
able  results  for  known  input  conditions.  This  is  very  similar  to  the  same  type  of  review 
described  as  a  component  of  verification.  In  the  former  case,  the  focus  of  the  SME  re¬ 
view  is  on  the  implementation  of  a  given  algorithm  in  the  code;  in  the  latter  case,  the  fo¬ 
cus  is  on  the  reasonableness  of  the  algorithm’s  results  for  a  given  application.  The 
boundary  between  this  aspect  of  verification  and  face  validation  is  indistinct. 
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Many  model  users  cited  data  validation  as  a  major  consideration  in  any  accreditation 
decision.  Data  validation  is  ensuring  that  the  data  to  be  used  in  a  study  are  representa¬ 
tive  of  the  real  world.  Since  there  are  numerous  data  sources  for  each  model,  data  se¬ 
lection  is  uniquely  dependent  on  each  particular  application.  Data  validation  is  most  ef¬ 
ficiently  done  by  each  model  user. 

4.3  Information  Elements  Supporting  Configuration  Management 

Configuration  management  is  also  a  significant  concern  of  the  study  participants. 
Adequate  configuration  control  is  needed  to  assure  the  model  user  that  the  version  of 
the  model  being  used  matches  documented  characteristics,  and  that  any  model 
changes  from  the  documented  version  are  identified.  A  configuration  management 
system  will  also  allow  the  accreditation  proponent  to  relate  past  accreditation  and  usage 
information  to  a  particular  model  version,  and  to  assess  the  applicability  of  that  historical 
information  to  the  current  application. 

Configuration  management  requirements  are  also  specified  in  the  Army’s  AR  5-11  and 
the  draft  Navy  VV&A  principles  (see  companion  volume  of  this  report  for  a  discussion  of 
these  directives).  The  draft  Air  Force  policy  did  not  address  configuration  management. 
However,  the  policy  is  undergoing  multiple  reviews  and  will  likely  be  modified  to  include 
configuration  management  requirements.  All  of  the  configuration  management 
requirements  are  focused  on  having  the  ability  to  track  model  changes  and  ensure  that 
the  V&V  results  are  applicable  to  the  version  of  the  model  used  in  a  particular  appli¬ 
cation. 

4.4  Information  Elements  Supporting  Accreditation 

In  addition  to  verification  and  validation  results,  information  in  other  categories  must  be 
collected  and  reviewed  to  satisfactorily  accredit  a  model.  This  information  falls  into  four 
categories:  model  descriptions;  model  documentation;  configuration  management  data; 
and  V&V  documents.  For  any  accreditation  decision,  a  clear  description  and  under¬ 
standing  of  the  model  to  be  used  is  essential.  This  description  includes  several  sub¬ 
elements  as  shown  in  the  information  requirements  matrix.  Model  features  must  be 
known  and  clearly  understood  to  permit  a  comparison  with  acceptance  criteria.  Model 
V&V  reports  produced  by  the  SMART  project  will  provide  nearly  all  the  necessary  infor¬ 
mation  to  meet  these  needs. 

The  availability  and  adequacy  of  model  documentation  is  an  area  of  concern  to  an  ac¬ 
crediting  authority.  The  most  commonly  identified  manuals  required  for  accreditation 
are  the  User's  Manual  and  the  Analyst's  Manual.  According  to  a  number  of  study  partic¬ 
ipants,  these  manuals  are  the  very  minimum  required  to  understand  a  model  and  use  a 
model  knowledgeably  in  a  particular  application.  The  documentation  review  that  is  con¬ 
ducted  as  part  of  the  SMART  process  ensures  that  these  documents  meet  the  minimum 
needs  of  a  model  user  by  comparing  the  current  status  of  model  documentation  with 
identified  accreditation  requirements. 

Plans  and  reports  that  document  VV&A  efforts  are  required  by  many  of  the  organiza¬ 
tions  that  have  formalized  accreditation  processes.  A  formal  accreditation  report  was 
the  most  frequently  cited  document  needed  to  support  accreditation.  Two  of  the  organi- 
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zations,  PMA-281  and  the  Navy  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 
desire  or  require  an  accreditation  report,  since  they  rely  on  external  sources  for  accredi¬ 
tation  recommendations.  OPTEVFOR  identifies  the  models  they  intend  to  use  and  re¬ 
quests  an  accreditation  study  on  those  models  from  the  Center  for  Naval  Analyses.  The 
respondent  report  serves  as  the  basis  for  an  accreditation  decision  by  the  Commander, 
OPTEVFOR.  PMA-281,  the  Cruise  Missile  Project  Office,  does  not  have  anyone  per¬ 
forming  formal  accreditation  studies.  Flowever,  they  would  like  to  have  such  a  study 
done  on  the  models  they  use  so  that  the  analytical  results  would  be  more  credible. 

Besides  the  general  accreditation  support  information  discussed  above,  other  informa¬ 
tion  was  identified  as  required  to  support  accreditation  either  for  a  class  of  applications 
or  a  specific  application.  The  most  frequently  required  information  to  support  a  class  of 
applications  (or  domain)  accreditation  decision  is:  V&V  Documentation,  Accreditation  & 
Usage  Histo ry,  and  Acceptance  Criteria.  The  first  two  elements  are  important  to  users 
because  they  provide,  in  summary  form,  a  broadly  based  statement  of  confidence 
regarding  the  model  along  with  a  definition  of  the  applications  for  which  the  model,  was 
deemed  suitable.  These  elements  can  also  provide  leads  for  sources  of  additional  V&V 
information  that  might  be  helpful  in  accrediting  the  model  for  a  specific  application.  The 
Accreditation  Support  Database  being  developed  by  the  SMART  Project  will  provide  the 
vast  majority  of  this  information. 

The  third  information  element,  Acceptance  Criteria,  is  considered  the  most  important 
factor  in  supporting  an  accreditation  decision,  considering  the  number  of  categories  in 
which  this  one  element  is  cited  as  a  requirement.  Acceptance  criteria  are  needed  for 
both  domain  and  application  specific  accreditation,  as  well  as  evaluating  validation  re¬ 
sults.  Their  importance  is  also  supported  from  a  logical  perspective.  Without  well- 
defined  acceptance  criteria,  any  judgment  concerning  model  suitability  for  an  application 
is  purely  subjective.  Acceptability  criteria  are  the  key  information  element  required  to 
reach  any  accreditation  decision.  A  list  of  sample  questions  that  might  be  used  to  guide 
the  selection  of  acceptance  criteria  is  provided  in  Appendix  E. 

There  are  two  potential  risks  in  not  defining  acceptability  criteria  and  relying  on  subjec¬ 
tive  judgment  of  model  suitability.  The  first  is  that  the  model  may  not  accurately  predict 
real  world  outcomes  throughout  the  envelope  of  interest  and  yet  it  might  be  accredited, 
thus  yielding  erroneous  study  results.  In  this  case,  major  programmatic  and  funding 
decisions  may  be  in  error,  resulting  in  a  waste  of  both  time  and  money.  Such  errors  can 
be  minimized  or  eliminated  through  clear  definition  of  acceptability  criteria  that  clearly 
delineate  the  boundaries  within  which  the  M/S  must  perform  acceptably.  A  second,  and 
more  likely  possibility  is  that  a  model  might  undergo  excessive  V&V  in  order  to  be  made 
as  perfect  as  possible,  leading  to  excessive  accreditation  costs.  A  clear  definition  of 
acceptance  criteria  gives  greater  confidence  that  an  M/S  can  be  accredited  without  an 
excessive  expenditure  of  funds  to  create  a  near  perfect  replication  of  a  real  system  or 
phenomenon. 

To  accredit  a  model  for  a  specific  application,  the  most  frequently  cited  information  re¬ 
quirement  besides  acceptance  criteria  was  the  correlation  of  scenario  and  input  data 
with  the  model's  requirements.  The  comparison  of  the  scenario  and  data  elements  of 
the  particular  application  with  those  of  the  model  helps  ensure  that  the  model  addresses 
all  of  the  elements  of  the  real  world  that  apply  to  the  problem  being  analyzed.  It  is  also 
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the  basis  for  comparing  the  model  limitations  to  the  problem  boundaries  to  ensure 
accurate  model  results  throughout  the  problem  domain. 
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5.  SMART  SOURCES  FOR  REQUIRED  INFORMATION 


Any  information  elements  cited  as  being  important  for  verifying,  validating,  and  accredit¬ 
ing  a  model  are  listed  in  Table  1  and  are  repeated  in  Table  3  to  show  correlation  with 
the  SMART  products.  The  second  column  in  Table  3  identifies  the  SMART  product  that 
contains  the  indicated  type  of  information.  Some  of  the  information  elements  are  not 
currently  included  in  the  SMART  products  but  can  be  added  with  relatively  little  addi¬ 
tional  effort;  the  third  column  shows  which  products  could  be  modified  to  include  the 
indicated  information. 

Some  of  the  information  can  only  be  derived  from  the  analytical  problem  (e.g.,  accept¬ 
ability  criteria,  real  world  descriptions,  scenario  descriptions,  etc.).  Such  information 
must  be  obtained  by  the  analyst  responsible  for  the  study  through  an  analysis  of  the 
study  requirements,  MOEs,  and  objectives.  Those  elements  are  identified  as  the 
“analyst’s  responsibility.” 

Other  information  elements  are  not  considered  important  by  a  wide  number  of  users,  or 
are  not  applicable  to  the  susceptibility  models  that  are  the  subject  of  the  SMART  pro¬ 
cess.  In  those  cases  the  element  is  shown  as  “not  addressed.”  An  explanation  of  the 
rationale  for  each  of  the  third  column  entries  follows. 

5.1  Information  Requirements  Supporting  Verification 

The  verification  process  undertaken  as  a  part  of  the  SMART  process  already  includes 
the  most  frequently  mentioned  and  most  beneficial  verification  steps.  These  are  a 
check  of  assumptions  and  constraints,  algorithm  logic  checks,  documentation  reviews, 
design  logic  checks,  sensitivity  analysis,  automated  test  tools  use,  and,  most 
importantly,  code  walk  throughs.  During  the  logic  checks,  a  units  check  is  made  to 
ensure  that  proper  units  of  measure  are  specified  and  are  consistent  with  the  units  in 
the  model  equations.  The  SMART  products,  as  presently  constituted,  provide  a 
significant  amount  of  the  most  frequently  needed  verification  information.  That 
information  not  included  is  discussed  in  the  following  paragraphs. 

The  Input  Data  Interfaces  Check  is  a  review  of  input  data  sources  for  correctness  from 
the  perspective  of  the  intended  application.  It  also  includes  a  check  for  consistency  of 
definitions  and  units  of  measure  between  the  data  sources  and  the  model.  This  verifi¬ 
cation  technique  requires  a  knowledge  of  the  data  sources  and  the  particular  applica¬ 
tion.  Therefore  it  must  be  done  by  the  analyst  for  each  application.  It  is  not  appropriate 
or  feasible  to  perform  this  type  of  verification  universally  for  all  applications.  Thus 
SMART  does  not  include  the  results  of  this  particular  technique  in  any  product. 
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TABLE  3  SMART  PRODUCT  CORRELATION  (Sheet  1) 


Verification  Reports 


Verification  Reports 


Verification  Reports 


Verification  Reports 


Verification  Reports 


Validation  Reports 


Verification  Reports 


Verification  Reports 


Verification  Reports 


ACTIVITY 


VERIFICATION  REQUIREMENTS 


Logical  Verification 


Rev  Assumptions/Constraints 


Algorithm  Logic  Checks 


Documentation  Review 


Input  Data  Interfaces  Check 


Design  Logic  Checks 


Peer  Review 


Code/Spec  Comparisons 


Verify  M/S  Specification 


Time/Step  Size  Impacts 


Code  Verification 


Sensitivity  Analysis 


Automated  Test  Tools 


Algorithm  Code  Checks 


Code  Walk  Throughs 


Stress  Testing 


Units  Check 


Internal  Consistency  Checks 


Quality  Verification 


Acceptance  Testing  -  Verification 


Mathematical  Stability  Check 


Stochastic  Model  Statistic  Tests 


Rule  Based  Systems  Tests 


VALIDATION  REQUIREMENTS 


Real  World  Applications  Defined 


Key  Application  Criteria  Defined 


Data  Comparison 


Benchmarking 


Check  Assumptions/Constraints 


Data  Validation 


Face  Validation 


Process  Sample  Data 


Independent  Review 


Sensitivity  Analysis  -  Stress  Test 


Any  Accredited  Algorithms 


Functional  Decomposition  Validation  Report 


Identify  Non  Validated  Model  Elements 


FE  Correlation  with  Manufacturer's 
Data 


Correlation  with  Intelligence  Data 


CURRENT 

RECOMMENDED 

SMART  SOURCE 

SMART  SOURCE 

Validation  Report 


Validation  Report 


Not  Addressed 


Not  Addressed 


Not  Addressed 


Not  Addressed 


Analyst's  Responsibility 


Analyst's  Responsibility 


Validation  Report  (Based  on 
SURVIAC's  Practices) 


ASD 


Validation  Report 


Validation  Report 
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TABLE  3  SMART  PRODUCT  CORRELATION  (Sheet  2) 


ACTIVITY 


ACCREDITATION  REQUIREMENTS 


Model  Description 


Model  Purpose/Usage 


In/Out  Data  Requirements 


Model  Fidelity 


Scenarios  Included/Excluded. 


Variables  Included/Excluded 


HW/SW  Requirements 


Model  Development  History 


Derivative  Analysis 


Model  Documentation 


User's  Manual 


Analyst's  Manual 


Programmer's  Guide 


Data  Dictionary 


Source  Code  Documents 


Test  Plans/TEMP 


Executive  Overview 


Configuration  Management 


CM  Status  Accounting 


CM  Internal  Control 


CM  External  Control 


User's  Group 


Configuration  Management  Plan 


Statement  of  Functional  Changes 


VV&A  Documentation 


Accreditation  Report 


Accreditation  Plan 


V&V  Plan 


V&V  Report 


ACCREDITATION  REQUIREMENTS 


Accreditation  &  Usage  History 


V&V  Documentation  Check 


Establish  Accreditation  Criteria 


Review  Assumptions  &  Constraints 


ACCREDITATION  REQUIREMENTS 


Scenario/Data  Inputs  Check 


Specific  Acceptance  Criteria 


Operator/Analyst  Expertise 


Independent  Review  Group 


CURRENT 

RECOMMENDED 

SMART  SOURCE 

SMART  SOURCE 

(GENERAL) 


ASD 


Model  Documentation 


VV&A/CM  Status  Report 


Model  Documentation 


Model  Documentation 


Model  Documentation 


ASD 


Not  Addressed 


Not  Addressed 


Documentation  Report 


Documentation  Report 


Documentation  Report 


Post  Development  Design 
Document 


VV&A/CM  Status  Report 


CM  Plan 


CM  Plan 


CM  Plan 


CM  Plan 


CM  Plan 


Not  Addressed 


Not  Addressed 


SMART  Process  Descriptions 


VV&A/CM  Status  Report 


(CLASS  OF  APPLICATIONS) 


Verification  Reports 


(SPECIFIC  APPLICATION) 


ASD 


ASD 


Analyst's  Responsibility 


ASD 


Analyst's  Responsibility 


Analyst's  Responsibility 


Not  Addressed 


Not  Addressed 
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A  Peer  Review  requires  that  a  team  of  subject  matter  experts  review  the  logic  of  the 
model  algorithms  and  reach  a  consensus  that  the  model  represents  their  understanding 
of  the  real  world.  Although  the  SMART  process  includes  a  logical  verification  of  the 
model  algorithms  by  the  independent  verifier,  it  does  not  include  any  reviews  by  inde¬ 
pendent  subject  matter  experts.  It  is  felt  that  actual  comparisons  with  test  data  provide 
a  more  rigorous  means  of  ensuring  that  the  model  represents  the  real  world  and  fulfill  a 
larger  set  of  user  requirements  for  credible  models.  The  SMART  process  does  include 
an  end-to-end  assessment  of  the  model  validity  by  an  independent  validator.  This 
assessment  is  done  based  on  comparisons  between  model  outputs  and  real  world  data. 
The  results  of  this  end-to-end  assessment  are  reported  in  the  validation  reports.  Any 
results  of  previous  Peer  Reviews  would  be  included  in  the  SMART  Accreditation 
Support  Database  (ASD)  under  the  data  elements  related  to  previous  V&V  results. 

Time-step  Size  Impacts  Assessment  requires  that  the  analyst  check  that  the  step  size 
employed  by  the  model  is  fine  enough  to  detect  any  changes  of  interest  in  the  particular 
application.  This  comparison  can  only  be  done  based  on  an  understanding  of  the  real 
world  nature  of  the  intended  application.  Therefore  the  SMART  products  do  not  include 
any  time-step  size  impact  assessments. 

Three  code  verification  techniques  that  are  not  addressed  in  the  SMART  products  are: 
stress  testing,  acceptance  testing,  and  mathematical  stability  checks.  Stress  testing  is  a 
technique  whereby  the  model  is  run  with  sample  input  data  to  ensure  that  the  outputs 
are  realistic.  Input  data  is  generally  selected  at  the  boundary  conditions,  using  combi¬ 
nations  of  input  parameters  that  have  been  determined  to  be  critical  through  sensitivity 
analyses.  The  SMART  verification  process  does  not  include  this  type  of  technique. 
However,  the  model  developers,  in  performing  sensitivity  analyses  on  some  functional 
elements,  do  conduct  runs  using  boundary  parameters.  Their  purpose  is  to  show  that 
the  outputs  of  the  functional  element  are  reasonable.  They  report  the  results  of  such 
runs  as  face  validation  of  the  particular  functional  element. 

Acceptance  testing  is  an  independent  operational  check  of  the  model  software  by  the 
receiving  agency.  This  particular  technique  is  not  documented  separately  in  the 
SMART  products  since  it  is  the  responsibility  of  the  receiving  agency. 

Mathematical  stability  checks  are  special  model  runs  on  different  platforms  or  using 
varying  input  data  to  check  for  unstable  results.  Special  tests  using  different  platforms 
were  considered  immaterial  to  the  SMART  process,  given  the  responsibility  of  the 
configuration  manager  to  ensure  platform  compatibility. 

The  other  two  verification  techniques  that  are  not  included  in  the  SMART  process, 
stochastic  model  statistic  tests  and  rule  based  systems  tests  do  not  apply  to  the  types  of 
models  addressed  by  the  SMART  Project  to  date.  However,  some  air  combat  models, 
such  as  TAC  BRAWLER,  might  benefit  from  performing  rule  based  systems  tests.  If  so, 
the  results  of  such  tests  will  be  included  in  the  validation  reports. 

The  final  verification  technique  that  is  not  specifically  included  in  the  SMART  process  is 
an  Internal  Consistency  Check.  This  is  a  check  that  consistent  units  of  measure  for 
time,  distance,  and  spatial  coordinates  are  used  throughout  the  model.  Since  the 
SMART  process  is  based  on  doing  V&V  by  functional  element,  a  check  for  internal  con- 
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sistency  in  the  total  model  is  not  done.  However,  as  each  functional  element  is  verified, 
the  code  check  does  address  internal  consistency  within  that  functional  element.  As 
each  additional  functional  element  is  verified,  consistency  with  units  of  the  previous 
functional  elements  should  also  be  checked.  In  this  way,  internal  consistency  will  be 
assured  once  the  total  model  has  undergone  a  code  check.  The  independent  verifiers 
should  specifically  report  the  results  of  internal  consistency  checks  on  each  functional 
element  so  that  subsequent  verifiers  can  build  upon  previous  work  and  eventually  check 
that  units  are  consistent  throughout  the  model. 

5.2  Information  Requirements  Supporting  Validation 

The  two  most  important  factors  related  to  validation,  Definition  of  the  Real  World 
Application  and  Identification  of  the  Application  Criteria,  are  dependent  on  each  particu¬ 
lar  application.  Therefore,  these  steps  are  the  analyst’s  responsibility.  The  SMART 
documents  do  list  the  intended  applications  for  a  model  that  can  be  compared  to  the 
analyst’s  concept  of  the  particular  application  to  lead  to  a  determination  that  a  model  is 
suitable  for  a  particular  application. 

With  clear  definitions  of  the  real  world  conditions  and  the  criteria  that  the  model  must 
meet,  the  actual  comparison  of  model  output  with  real  world  data  can  be  done.  This 
step  is  the  heart  of  the  SMART  process  and  is  one  of  the  fundamental  results  of  apply¬ 
ing  the  SMART  process  to  a  model.  This  step  is  looked  upon  as  the  best  means  of  vali¬ 
dating  an  engineering  model  since  one-to-one  comparisons  under  controlled  conditions 
can  be  made.  The  SMART  products,  as  currently  constituted  provide  this  information 
and  are  a  valuable  source  of  information  to  a  model  user. 

Benchmarking,  while  not  a  part  of  the  SMART  process,  will  be  addressed  in  the  SMART 
Accreditation  Support  Database  (ASD).  Any  results  from  model  user  benchmark  com¬ 
parisons  will  be  listed  as  part  of  the  V&V  history  of  a  model.  The  ASD  will  include  a 
listing  of  the  functional  elements  that  have  been  validated.  Through  a  simple  compar¬ 
ison,  those  not  validated  can  also  be  identified,  thus  providing  another  piece  of  desired 
information. 

Two  of  the  validation  elements  are  not  addressed:  Data  Validation  and  Identification  of 
Any  Accredited  Algorithms.  Data  Validation  is  a  check  on  input  data  sources  to  ensure 
that  the  data  are  representative  of  the  real  world.  Although  the  SMART  Project  certifies 
its  own  validation  data  sources,  data  to  be  used  in  other  studies  will  vary  for  each  appli¬ 
cation  and  the  types  of  systems  may  vary  from  application  to  application.  Data 
Validation  in  this  context  is  best  done  by  the  analyst  performing  the  study.  Identification 
of  Accredited  Algorithms  is  a  check  to  determine  if  any  of  the  algorithms  used  in  the 
model  have  been  previously  verified  or  validated  in  other  models.  Such  a  check  is 
useful  if  one  is  unfamiliar  with  a  model  and  just  starting  a  V&V  effort,  but  considering  the 
extended  V&V  performed  under  the  SMART  process,  this  technique  is  not  addressed. 
As  part  of  the  SMART  process,  a  V&V  status  survey  of  the  models  in  its  domain  was 
conducted  and  little  or  no  documented  work  of  recent  vintage  was  found. 

Information  on  the  three  remaining  validation  elements  should  be  included  in  the 
SMART  validation  reports  when  feasible.  The  Survivability  &  Vulnerability  Information 
Analysis  Center  (SURVIAC)  processes  sample  data  through  a  model  when  it  is 
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released  to  them  for  distribution.  These  runs  with  sample  data  check  that  a  new  version 
of  a  model  provides  the  same  results  as  the  previous  version  for  standard  inputs.  This 
check  provides  assurance  that  any  modifications  did  not  introduce  unintentional 
changes  in  parts  of  the  model  that  were  to  remain  the  same.  It  would  appear  to  be  easy 
to  incorporate  the  results  of  these  runs  into  the  validation  report  at  one  of  the  periodic 
updates.  These  results  could  also  be  summarized  in  the  ASD,  which  SURVIAC  will 
administer. 

The  functional  element  correlation  with  manufacturer’s  data  is  applicable  to  U.S.  or  al¬ 
lied  systems  for  which  manufacturer’s  data  is  accessible.  Since  the  SMART  process  is 
based  on  validation  using  actual  data  obtained  from  these  systems,  this  specific  tech¬ 
nique  provides  little  additional  value.  Such  comparisons  would  be  valuable  as  part  of  a 
face  validation.  Any  results  from  such  comparisons  that  are  made  as  part  of  face  vali¬ 
dation  for  selected  functional  elements  will  be  included  in  the  appropriate  Functional 
Element  Assessment  Reports. 

Correlation  with  intelligence  data  is  a  specific  requirement  of  the  J-MASS  and  SIMVAL 
programs.  The  results  from  those  programs  that  bear  on  the  models  in  the  SMART 
domain  will  be  included  in  the  ASD.  Any  intelligence  correlation  results  from  on  going 
parallel  SIMVAL  efforts  should  be  obtained  and  summarized  in  the  VV&A/CM  Status 
Reports  and  the  ASD. 

With  these  additions,  the  SMART  products  can  become  a  single  source  of  all  or  nearly 
all  V&V  information  to  support  accreditation  for  models  it  is  assessing.  The  only 
information  not  included  will  be  V&V  results  for  model  changes  made  by  the  user  or  for 
specialized  scenario  conditions  not  included  in  the  baseline  data. 

5.3  Information  Requirements  Supporting  Configuration  Management 

The  SMART  Project  is  developing  a  common  configuration  management  plan  for  all  the 
models  that  will  undergo  the  SMART  V&V  process.  This  plan  will  meet  all  the  major 
C/M  information  requirements  identified  in  this  study.  A  prototype  configuration  man¬ 
agement  process  will  be  established  and  tested  on  one  of  the  SMART  models  during 
FY94.  The  prototype  system  will  be  used  to  track  different  model  versions  and  relate 
the  V&V  work  done  by  both  the  SMART  Project  and  others  users  to  each  version. 

“Based  on  the  results  of  this  prototype  effort  a  common  C/M  process  will  be  proposed 
for  use  on  a  continuing  basis  for  all  three  SMART  models.  This  system  will  meet  the 
needs  of  all  the  model  users  interviewed  for  this  study.  The  common  C/M  process  will 
be  documented  in  a  C/M  procedures  guide. 

5.4  Information  Requirements  Supporting  Accreditation 

Model  descriptive  information  is  currently  contained  in  the  model  documentation.  Some 
of  it,  such  as  the  purpose  and  usage,  will  be  in  the  ASD.  Other  elements,  such  as  a  list¬ 
ing  of  the  intended  scenarios  and  the  hardware  and  software  requirements  should  also 
be  added.  The  items  related  to  the  model  development  history  and  its  derivation  from 
other  models  is  not  in  any  formal  SMART  report,  but  is  frequently  a  part  of  the  model 
documentation.  The  documentation  assessment  requirements  developed  by  the 
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SMART  Project  do  require  that  the  model  history  be  included  in  one  of  the  manuals 
produced  by  the  model  developer.  Any  specific  deficiencies  in  the  model  history  are 
documented  in  the  individual  model  Documentation  Assessment  Reports  that  are 
produced  as  part  of  the  verification  process. 

The  three  principle  model  documents  (User’s  Manual,  Analyst’s  Manual,  and 
Programmer’s  Manual)  are  reviewed  and  evaluated  as  part  of  the  SMART  verification 
process.  These  reviews  will  help  ensure  that  these  manuals  meet  a  minimum  standard 
of  acceptability  and  applicability  to  accreditation  based  on  identified  requirements. 
Furthermore,  based  on  information  from  these  reviews  and  inputs  from  the  model 
developer,  a  Post  Development  Design  Document  is  developed.  This  document  serves 
the  purpose  of  the  developer’s  software  specification  for  verification  purposes. 
Summaries  of  the  contents  and  quality  of  these  documents  are  provided  in  the 
Documentation  Assessment  Report.  The  Data  Dictionary  is  not  addressed  separately  in 
the  SMART  products.  The  documentation  assessment  requirements  do  include  a 
requirement  that  adequate  descriptions  of  input  and  output  data  be  included  in  one  of 
the  three  manuals  or  a  separate  document.  Test  Plan  reviews  are  not  included  in  the 
SMART  process.  Test  plans  that  address  model  usage  are  unique  to  each  application 
and  therefore  the  responsibility  for  checking  these  test  plans  rests  with  the  individual 
analyst. 

The  information  that  supports  accreditation  for  a  class  of  applications:  namely,  accredi¬ 
tation  and  usage  history,  V&V  documentation  lists,  and  lists  of  assumptions  and  con¬ 
straints,  will  be  included  in  the  ASD.  The  assumptions  and  constraints  are  currently 
contained  in  the  verification  reports  in  greater  detail.  Accreditation  or  acceptance  crite¬ 
ria  depend  on  the  intended  application  and  are  the  analyst’s  responsibility. 

The  information  requirements  for  application  specific  accreditation  either  depend  on  the 
intended  application  or  are  outside  the  scope  of  the  SMART  Project.  They  are  not  ad¬ 
dressed  in  any  of  the  SMART  products. 

6.  SMART  PRODUCT  USAGE 

The  V&V  data  produced  through  the  application  of  the  SMART  process  form  a  library  of 
standard  information  that  is  needed  to  support  nearly  all  accreditations.  Use  of  this 
library  should  simplify  the  task  of  developing  accreditation  support  packages.  Adoption 
of  the  SMART  process  for  additional  V&V  beyond  the  scope  of  the  existing  data  will 
improve  the  breadth,  depth  and  quality  of  the  V&V  work  done  and  increase  the  overall 
efficiency  of  accreditation  efforts. 

The  SMART  Project  is  developing  a  library  of  V&V  information  for  selected  models. 
Since  the  SMART  process  is  applicable  to  any  engineering  type  model,  the  benefits 
could  be  realized  if  the  users  of  a  particular  model  all  adopted  this  process  for  their  V&V 
work.  Since  the  amount  of  information  generated  by  application  of  the  SMART  process 
is  extensive,  development  of  the  baseline  information  library  requires  sufficient  time  and 
resources.  Individual  users  often  will  not  have  sufficient  amounts  of  either  to  undertake 
the  foundation  work. 
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To  realize  the  benefits  of  the  SMART  process  without  unduly  burdening  a  single  user, 
the  process  can  be  divided  into  a  series  of  incremental  steps,  each  of  which  provides  a 
partial  contribution  to  the  baseline  library  and  fulfills  some  of  the  information 
requirements  identified  in  this  study.  This  incremental  process  includes  enhancements 
to  the  current  SMART  process  that  provide  some  initial  credibility  assessment  prior  to 
full  validation  and  verification.  These  enhancements  include  a  peer  review  to  verify  the 
adequacy  of  model  algorithms  and  face  validation  of  the  whole  model.  The  incremental 
approach  to  applying  the  SMART  process  is  depicted  in  Figure  1.  This  approach  would 
also  be  useful  for  legacy  models  not  included  in  the  initial  SMART  domain. 

The  diagram  is  arranged  so  that  the  steps  are  in  sequential  order  from  top  to  bottom. 
Those  steps  which  are  horizontally  parallel  would  generally  be  performed  concurrently. 
If  sufficient  resources  were  not  available,  the  user  could  choose  which  of  the 
horizontally  aligned  steps  to  perform  based  on  individual  needs.  The  arrows  show 
which  steps  must  be  performed  to  yield  the  indicated  products.  The  process  support 
products  are  a  natural  by-product  of  the  SMART  process  application  and  facilitate 
execution  of  subsequent  steps. 

The  overall  process  is  structured  so  that  each  V&V  product  provides  an  incremental 
increase  in  model  credibility.  Each  model  user  can  perform  as  many  steps  as  time  and 
resources  permit.  The  scheme  for  performing  the  V&V  and  organizing  the  resulting 
information  is  discussed  further  in  the  companion  report  entitled  “A  Comparative 
Analysis  of  Tri-Service  Accreditation  Policies  and  Practices,”  Volume  I  of  Report  # 
JTCG/AS-93-SM-20.  Ultimately,  when  all  of  the  functional  elements  (FEs)  are  both 
verified  and  validated,  and  overall  model  validation  is  completed  and  correlated  with  FE 
validation  as  appropriate,  a  library  of  baseline  V&V  data  will  exist  that  will  provide  nearly 
all  of  the  supporting  information  for  subsequent  accreditations.  With  such  a  library  of 
information,  each  model  user  need  only  perform  additional  V&V  on  changed  portions  of 
a  model  or  for  unique  scenarios.  If  this  additional  information  is  then  fed  back  into  the 
library,  each  subsequent  user  has  a  greater  body  of  information  on  which  to  base 
accreditation. 

To  make  the  V&V  information  collected  in  this  process  most  useful,  results  would  be 
stated  in  objective  terms  so  that  individual  model  users  can  make  independent 
judgments  regarding  model  suitability  for  a  given  application.  Subjective  terms  such  as 
“good  correlation”  or  “compares  favorably”  would  be  avoided.  Where  possible,  results 
of  any  comparison  between  model  predictions  and  real  world  data  would  be  quantified. 
Adherence  to  this  philosophy  would  enhance  the  utility  of  validation  data  collected  by 
other  users.  The  SMART  Project  is  currently  undergoing  a  detailed  review  of  the  format 
of  its  accreditation  support  products  to  provide  baseline  examples  of  how  such 
information  should  be  structured. 
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FIGURE  1  INCREMENTAL  SMART  PROCESS 


23 


7.  CONCLUSIONS  AND  RECOMMENDATIONS 


A  principal  finding  of  this  study  is  that  all  the  information  produced  by  the  SMART  pro¬ 
cess  is  important  to  most  accreditation  decisions.  Another  finding  is  that  the  SMART 
process,  as  presently  structured,  produces  all  of  the  non-application-unique  information 
most  frequently  cited  as  required  to  support  accreditation.  If  the  recommended 
additions  that  are  listed  below  are  incorporated,  the  SMART  products  will  contain  over 
70%  of  all  the  information  that  was  mentioned  in  this  study.  Information  that  is  not 
included  was  deemed  of  marginal  value  since  it  was  mentioned  by  only  one  or  two 
sources  or  is  redundant  with  other  V&V  information  produced  by  the  SMART  process. 
Once  the  SMART  process  has  been  applied  to  a  model  to  generate  a  library  of  V&V 
data,  a  user  will  only  need  to  perform  additional  V&V  for  model  changes  or  for  unique 
conditions  of  the  intended  application. 

Use  of  the  SMART  products  will  save  the  typical  user  a  significant  amount  of  time  and 
money  that  would  normally  be  spent  in  collecting  or  producing  extensive  V&V 
information.  Employment  of  the  SMART  process  in  developing  application  specific  V&V 
results  will  also  minimize  the  amount  of  potential  variability  that  exists  in  the  quality  and 
scope  of  V&V  that  is  done  and  will  provide  additional  information  in  a  standard  format 
for  future  users. 

To  realize  the  benefits  of  having  a  library  of  standard  V&V  data,  and  to  make  that  data 
most  useful,  the  following  recommendations  are  submitted. 

1.  The  processes  for  determining  study  MOEs  and  MOPs  outlined  in  the  com¬ 
panion  volume  to  this  report  should  be  adopted. 

2.  V&V  results  from  the  SMART  Project  should  be  stated  in  objective  terms 
relating  to  model  limitations  and  constraints. 

3.  SMART  reports  and  the  ASD  should  be  tailored  to  include  the  information 
elements  indicated  in  Table  3.  Specifically,  the  following  types  of  data  should 
be  obtained  and  included  in  the  SMART  products: 

•  Internal  Consistency  Check  results  for  each  functional  element  should 
be  reported  in  the  verification  reports  to  establish  a  means  of  checking 
that  the  model  uses  consistent  units  throughout. 

•  Results  of  SURVIAC  sample  data  runs  should  be  included  in  the 
validation  reports  to  ensure  that  new  versions  of  a  model  produce 
expected  results  for  standard  inputs.  These  results  build  confidence 
that  model  changes  did  not  introduce  errors. 

•  Functional  element  correlation  with  manufacturer’s  data  (when  such 
data  iare  available)  should  be  used  for  validation  if  actual  test  data  is 
not  available.  Such  correlation  will  provide  almost  as  much  confidence 
in  model  realism  as  test  data  comparisons. 
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•  Any  results  of  correlation  with  approved  intelligence  data  (e.g., 
SIMVAL  results)  should  be  included  in  the  validation  reports.  Inclusion 
of  these  results  will  provide  all  of  the  V&V  history  in  one  source  and 
facilitate  each  accreditation. 

4.  The  model  managers  for  each  of  the  models  in  the  SMART  domain  should 
encourage  their  model  users  to  adopt  the  SMART  process  for  any 
application-specific  V&V.  Any  V&V  results  should  be  provided  to  SURVIAC 
for  incorporation  into  the  SMART  Accreditation  Support  Database. 

5.  The  results  of  this  study,  relating  to  accreditation  support  information  require¬ 
ments,  should  be  reviewed  in  one  year  to  determine  any  changes  that  might 
evolve  due  to  changing  service  policies. 
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APPENDIX  B 


INTERVIEW  GUIDE 


ACCREDITATION  REQUIREMENTS  STUDY 
INTERVIEW  GUIDE 


1  For  what  purpose(s)  does  your  agency  use  M&S? 

2  What  is  your  accreditation  process? 

3  Does  it  conform  to  any  of  the  process  models  we  have  already  diagrammed? 
Which  one? 

4  Who  has  authority  to  accredit  models  for  your  use? 

5  Is  that  authority  defined  in  any  instruction  or  charter? 

6  If  so,  what  is  that  document? 

7  Is  that  instruction/charter  agency-specific  or  service-wide? 

8  Can  we  have  a  copy? 

9  On  what  basis  are  accreditation  decisions  made? 

10  Considering  the  various  VV&A  elements,  (e.g.  Data  source  verification,  logical 
verification,  code  verification,  comparison  with  test  results,  comparison  with  other 
model  results,  etc.)  which  ones  are  required  for  your  accreditation?  Which  ones 
are  desirable? 

1 1  Is  validation  of  any  sort  required  prior  to  accreditation? 

12  Considering  the  validation  elements  already  listed,  which  ones  are  done?  Are 
there  any  others? 

13  Which  elements  would  you  consider  most  beneficial? 

14  Must  the  validation  have  been  done  on  the  version  of  the  model  being  accred¬ 
ited? 

15  Are  special  tests  run  to  validate  a  model,  or  are  previous  test  data  acceptable? 

16  For  the  last  few  models  accredited,  what  was  the  cost  of  any  special  testing  or 
data  collection  which  was  done? 

17  Have  any  models  been  accredited  solely  on  the  basis  of  past  usage,  or  on  the 
basis  of  “face  validation”? 

18  What  agencies  do  you  interface  with  to  obtain  information  on  VV&A  history  of  the 
model  you  intend  to  accredit? 

19  Is  any  verification  required  prior  to  accreditation? 
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20  Considering  the  verification  elements  already  listed,  which  ones  are  done  to  sup¬ 
port  an  accreditation  decision? 

21  Are  there  any  others? 

22  Is  configuration  management  of  a  model  required  prior  to  accreditation? 

23  To  what  extent? 

24  Is  configuration  management  handled  in-house,  or  does  your  agency  rely  on 
configuration  management  activities  from  other  agencies? 

25  Describe  any  configuration  management  requirements  imposed  on  the  accredita¬ 
tion  process,  and  any  procedures  used  to  fulfill  them. 

26  How  are  any  validation,  verification,  or  accreditation  activities  documented? 

27  Are  these  activities  and  results  published  in  any  form  by  which  other  model  users 
can  become  informed  of  the  results  and  use  them? 

28  What  are  these  forms,  and  can  we  have  access  to  some  typical  accreditation  re¬ 
ports? 

29  What  types  of  model  information  is  needed  to  determine  whether  the  model  ful¬ 
fills  the  analytical  requirement?  What  model  summary  data  would  you  want  in  a 
database? 

30  Do  you  have  any  examples  of  criteria  which  you  used  as  a  basis  for  making  an 
accreditation  decision  or  in  determining  that  a  model  was  acceptable  for  a  par¬ 
ticular  use? 

31  In  looking  at  the  matrix  of  accreditation  requirements,  which  ones  are  either  re¬ 
quired,  desired,  or  considered  acceptable  to  accredit  a  model. 

32  Who  else  in  this  agency  can  we  talk  to  concerning  accreditation  efforts? 

33  Who  else  in  your  service  should  we  be  talking  to  about  accreditation  require¬ 
ments? 
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AIR  FORCE  SPECIFIC  QUESTIONS 


For  AFOTEC 

34  Two  guides  exist  for  AFOTEC  M&S  accreditation;  POC:OAN  Draft  of  5  Feb.  92 
and  the  LG4  guide  of  Jan.  91.  Which  one(s)  express  AFOTEC  policy?  Who  is 
Accrediting  authority,  M/S  Committee  or  as  appointed  in  Accreditation  Plan? 

35  LG4  guide  says  analysts  are  responsible  for  Configuration  Control  for  M&S  they 
develop.  Does  AFOTEC  do  any  CM  for  existing  models? 

36  When  using  existing  models,  does  AFOTEC  require  that  the  four  documents 
(Management  manual,  Analysts  Manual,  Programmers  Manual,  &  Users  Manual) 
be  available? 

37  For  existing  models  if  manuals  do  not  meet  your  standards,  are  they  rewritten? 

38  Should  the  DOT&E  issues  (identified  on  the  separate  issues  sheet)  be  addressed 
or  investigated  as  part  of  the  SMART  process? 


NAVY  SPECIFIC  QUESTIONS 

39  What  relationship  exists  between  the  Navy  VV&A  plan  drafted  under  SPAWAR 
sponsorship  and  the  JACG  initiative  to  develop  a  unified  VV&A  process  for  sur¬ 
vivability  models? 

40  Of  the  SMART  model  set,  only  ESAMS  is  listed  in  the  M&S  catalogue  being  pre¬ 
pared  by  SPAWAR.  Should  the  others  be  included? 

41  Goal  #10  for  Team  MIKE  is  to  "establish  a  methodology/procedure  for  providing 
documentation,  certification,  and  configuration  management  for  Naval  Warfare 
models."  Do  the  SMART  documentation,  assessment  reports,  and  CM  plans 
meet  this  requirement? 

42  Goal  #7  is  to  "establish  and  catalog  common  data  bases  IAW  the  principles  sup¬ 
porting  Navy  evolution  toward  common  data  bases  through  the  NWTDB  process, 
set  forth  in  OPNAVINST  9410.5."  Need  more  information  about  that  instruction 
and  related  Navy  policies  on  data  base  development. 
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ARMY  SPECIFIC  QUESTIONS 


43  Is  there  an  Organization  Chart  which  shows  the  various  Army  organizations, 
committees,  and  activities  which  get  involved  in  accreditation? 

44  Any  contacts  in  Army  Model  and  Simulation  Management  Office  (AMSMO), 
Office  Deputy  Chief  of  Staff  for  Operations  (ODCSOPS),  or  MISMA  who  should 
be  interviewed? 

45  For  MISMA  What  progress  has  there  been  on  the  MISMA  master  plan 
(referenced  in  the  Jim  Metzger  brief)  for  credibility  assessment?  Is  that  AR  5-1 1  ? 

46  For  AMSMO  What  progress  on  the  AMSMO  "How  to"  handbook?  Who  is 
preparing  it?  Can  we  contact  them  for  information  gathering  purposes?  Who  is 
Point  of  Contact  (POC)? 

47  Are  accreditation  procedures  listed  on  pg.  6  of  SAUS-OR  memo  dated  30  Oct  89 
still  complete  &  valid?  What  does  the  statement  "Review  of  how  input  data  and 
scenario  data  are  used  or  modified  internal  to  the  model. "  mean? 

48  Can  I  get  a  sample  of  a  "good"  V&V  plan  and  V&V  report? 

49  For  OPTEC  Is  the  V&V  documentation  format  (end.  6  of  OPTEC  IPG  92-2)  still 
valid?  What  is  an  "uncertainty  analysis"? 

50  Are  any  SMART  models  included  in  the  list  of  Army  "Touchstone"  M&S?  What 
aircraft  survivability  models  are  included? 

51  AR  5-1 1  has  provisions  for  accreditation  of  a  model  for  a  "Class  of  Applications". 
Define/describe  a  "class  of  applications". 

52  Should  SMART  attempt  to  produce  a  V&V  report  supporting  accreditation  for  a 
Class  of  Applications?  If  so  what  class(es)  should  be  covered? 

53  Should  the  SMART  produced  test  data  conform  to  AR  25-9  and  the  Army  Data 
Encyclopedia?  If  so,  what  are  the  requirements  and  costs?  Would  Army  provide 
funds  to  support  tailoring  data  to  these  requirements?  Who  should  funding  re¬ 
quests  of  this  nature  be  directed  to? 

54  What  are  some  typical  acceptability  criteria  which  are  used  in  evaluating  a  model 
for  a  class  of  applications? 

55  Where  can  I  get  a  copy  of  AR  25-1  and  AR  25-9? 
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